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Color constancy (Toet and Franken, 2003 [2]; Toet, 2003 [7]) and color contrast (Scribner et al., 2000 [21]; 
Lee et al., 2005 [23]) are two important topics for color image fusion. The paper focuses on the low color 
contrast problem of linear fusion algorithms with color transfer method. Color transfer technology is pop¬ 
ular in infrared (IR) and visible image fusion to give the fused image a natural day-time color appearance 
(Toet, 2003 [7]; Wang et al., 2007 [8]; Zheng and Essock, 2008 [9]). However, in the color transfer step, all 
three channels of the color space are processed with the same linear mapping without color enhance¬ 
ment, resulting in low color contrast between the target and the background (Wang et al., 2007 [8]). 
Based on the characteristics of the IR image, we introduce a ratio of local to global divergence of the IR 
image to improve the color contrast. The enhancement ratios for both hot and cold targets are larger than 
one, while it tends to one for the background. As a result, the proposed method pops out both hot and 
cold targets in color, where hot targets will appear intense red, and cold targets will appear cyan. Subjec¬ 
tive results show visible color contrast enhancement effects. Target detection experiments through hue 
and saturation components of the fused image show an improvement in the hit rate for target detection, 
owing to larger color distance between the target and the background. 

Crown Copyright © 2009 Published by Elsevier B.V. All rights reserved. 


1. Introduction 

As two of the most commonly used image sensors in night-vi¬ 
sion for military applications, infrared (IR) and visible light cam¬ 
eras have their own advantages and limitations due to the 
physical differences between the image formation processes. The 
IR image records thermal radiations emitted by objects in a scene 
and can be utilized to discover targets as it has better hot contrast 
and can present camouflaged targets. The visible image has much 
more high-frequency information of the background, which is 
essential to accurate target localization and situation awareness 
[1,2]. Thus, appropriate fusion of IR and visible images can combine 
the complementary information and obtain a better description of 
the scene, which will help in target detection, target localization 
and situation awareness tasks [1-4]. 

Color acts as a basic factor in the human visual system. Percep¬ 
tual evaluation practices show that appropriately color fused 
night-vision imagery is often helpful for human observers to im¬ 
prove their performance, for instance, on situation awareness, tar¬ 
get detection, search and classification [1-4]. In recent years, color 
image fusion is becoming a more and more important research 
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field and lots of novel false color fusion approaches have been 
developed to fuse IR and visible images [5,6]. 

One key topic of the color fused imagery is color constancy. It 
has been shown that the lack of color constancy with inappropriate 
color mapping may hinder human performance [2]. In 2003, Toet 
originally introduced the color transfer technology to produce 
night-vision imagery with a natural appearance and with colors 
that are to some extent invariant for changes in the environmental 
conditions [7]. By transferring the mean and standard deviation of 
a selected natural day-time color imagery to the false color fused 
image for each channel in an appropriate color space, the approach 
can be used effectively to give multi-band night-vision imagery a 
natural day-time color appearance. Further more, it has been 
experimentally demonstrated that the false color fused image 
and the selected reference image do not have to be of the same 
scene [7-9]. Satisfactory results can be obtained so long as they 
are similar to each other to some degree, which extends applica¬ 
tions of the approach greatly. 

Another key topic of the color fused imagery is color contrast, 
which enhances human visual capabilities in discriminating fea¬ 
tures from their background and is essential to target detection 
tasks [21,23]. However, to date, little attention has been paid to 
the color contrast problem while implementing color transfer. In 
the color transfer step, all three channels of the color space are pro¬ 
cessed with the same linear mapping without color contrast 
enhancement [7]. As a result, targets that are clear in the IR image 
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may have the similar color to the background in the fused image 
and are difficult to detect [8]. In [9], Yufeng Zheng et al. proposed 
a novel local algorithm based on image segmentation, recognition 
and local color transfer to enhance the color mapping effect. How¬ 
ever, the color image segmentation and recognition are both time- 
consuming and inclined to make mistakes (85% of accuracy was re¬ 
ported). In this paper, we address the topic of color contrast and 
present a color contrast enhanced image fusion method to give 
boost to color of targets. 

2. Linear image fusion 

The schematic diagram of linear image fusion is shown in Fig. 1. 
The source IR and visible images are firstly fused by false color fu¬ 
sion and then the color transfer technology is applied to obtain a 
natural final fused image. 

Different false color fusion approaches and different color 
spaces can be utilized in the scheme [7-9]. In this paper, we imple¬ 
ment the false color fusion and color transfer step in YUV space, 
where Y denotes luminance, U stands for difference between blue 
and luminance, and V stands for difference between red and lumi¬ 
nance. The transform from RGB space to YUV space is written as: 
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The false color fusion in YUV space is carried out as: 

Y c = Vis, 

U c = Vis—IR, (2) 

V c = IR—Vis, 

where Vis and IR stand for the visible image and the IR image, 
respectively. By mapping the visible image to the achromatic chan¬ 
nel, the high-frequency information of the visible image is kept in 
the fused image. The differencing operation between the visible 
and the IR image provides intersensor contrast, which in a general 
sense is similar to the spatial center/surround color opponency 
shown in the primate visual system [10]. As a result, the chromatic 
channels of the fused image show intersensor contrast, making pop¬ 
ping targets in color possible. For the IR image, elements that are 
hotter (the human body for example) or colder (vehicles that keep 
still for example) than their background are deemed to be possible 
targets [11 ]. For the false color fusion approach adopted, as V is the 
difference between red and luminance, hot targets will appear red¬ 
dish. For cold targets, the green and blue channels will dominate in 
the final fused image, leading to the cyan color. 

In the following step, color transfer is implemented to make the 
final fused image and the reference image have the same mean and 
standard deviation for each channel in an appropriate color space. 
The color transfer technology is based on the fact that straightfor- 



Fig. 1 . Schematic diagram of linear image fusion. 


ward color statistics can capture some important subjective no¬ 
tions of style and appearance in images [12]. In YUV color space, 
the color transfer step can be written as: 

p F=$( p c-rt)+ri, p=y,u,v, (3) 

°c 

where the indexes C, R and F refer to the false color fused image, the 
reference image and the final fused image, respectively. P denotes 
an individual color channel of the image in YUV space, fi and a refer 
to the mean and standard deviation of channel P, respectively. By 
applying a linear mapping to each channel of the false color fused 
image as in Eq. (3), the final fused image will take the style and 
appearance of the reference image. 

3. Color contrast enhancement in IR and visible image fusion 

While the advantage of image fusion for situation awareness 
tasks has been well demonstrated [2,3], its effect on target detec¬ 
tion is still ambiguous as the detection mainly depends on contrast 
between the target and the background, which is usually high in 
the IR image [2-4]. Thus, ideal image fusion should maintain or im¬ 
prove the high detectability of the target in the IR image, while 
improving the situation awareness performance. 

Owing to the human visual system, however, the resolution 
ability to the intensity of a color image is worse than that of a gray 
image. Consequently, a higher hit rate for target detection can not 
be achieved by using only intensity information in the color image- 
fusion scheme. Fortunately, color contrast provides another cue in 
the color fused image, which makes the separation between the 
target and the background easier. It has been demonstrated that 
the human ability to detect small objects against a varied back¬ 
ground can be greatly facilitated by the use of color and fewer fix¬ 
ations are required to locate the target [13,14]. Further more, with 
large color contrast, the human performs a parallel target 
detection, while with low color contrast, response time increases 
with display density, indicating a serial search [15]. So, the color 
image fusion should pop out targets in color to improve their 
detectability. 

In the linear image-fusion scheme, the relationship between 
targets and colors is similar to that of color-coded images, i.e. the 
fused images are partially color-coded, which will help to enhance 
targets [16,17]. However, the color information of targets has not 
been utilized enough as no special method is taken to enhance 
its color contrast. As a result, targets may have the similar color 
to the background in the fused image and are difficult to detect. 

The main reason for low color contrast of the linear image-fu¬ 
sion scheme lies in that all three channels are processed with the 
same linear mapping in the color transfer step. To enhance the col¬ 
or contrast between the target and the background, we propose to 
enhance the V channel by a color contrast enhancement ratio rj\ 

{Pr = $(Pc-£) + l4, P=Y,U, 

U C 

< V F = ti.%(V c -n v c )+H v R , (4) 

n = D/n D , 

D = \IR~ n IR \, 

where D is the divergence of the intensity of each pixel from the 
mean intensity of the IR image, and rj is the ratio of local to global 
divergence of the IR image. A 3 x 3 window can be used to get the 
local mean divergence to eliminate noise in the IR image. 

As both hot and cold targets are of great divergences from the 
mean intensity of the IR image, both of them can be enhanced. 
With color contrast enhancement, hot targets will appear intense 
red, while cold targets will appear cyan in the fused image. The 
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red color of hot targets can alert observers of possible danger, and 
the cyan color can help observers to detect cold targets or low- 
temperature regions (if there are no cold targets). 

4. Fusion results 

To validate the effectiveness of our method, three image sets 
called uncamp , trees and dune are fused. For comparison, we also 
obtain the color fused images by linear image fusion without 
enhancement and by Wang’s [8] enhancement method. Three pairs 
of them are completely analyzed and shown in Figs. 2-4 (namely, 
NV1816, NV4917 and NV7404). Subjective results show visible col¬ 
or contrast enhancement between the target and the background 
in the final fused image. 

For the NV1816 pair shown in Fig. 2, the man is the hot target, 
and the top region of the house is a cold target in the IR image. A 
natural day-time color image shown in Fig. 2c is selected as the ref¬ 
erence image. As analyzed in Section 2, the man and the dark re¬ 
gion should appear red and cyan respectively in the fused image. 
In Fig. 2d, however, without color contrast enhancement, the 
man takes on almost the same color as the background and is very 
difficult to detect. Wang’s enhancement method gives considerable 
boost to color of the hot target as shown in Fig. 2e. However, little 
temperature information of the cold region is provided through 
color. By the proposed method, the man appears intense red and 
the cold target appears deep cyan, which is in accordance with pre¬ 
vious analysis in Section 3 and can pop out both targets. At the 
same time, details of the background are well kept as can be seen 
in Fig. 2f. 

The same conclusion can be deduced from experiments using 
NV4917 and NV7404 pairs. As we can see in Figs. 3 and 4d, without 
enhancement the man is hard to distinguish from the background. 
Wang’s method provides a better result as the human appears red, 
despite part of it is still covered by the green color. The man in the 


Fig. 2. IR and visible image fusion of image pair NV1816: (a) visible image; (b) IR 
image; (c) reference image; (d) result without enhancement; (e) result by Wang’s 
method; (f) result by our method. 


Fig. 3. IR and visible image fusion of image pair NV4917: (a) visible image; (b) IR 
image; (c) reference image; (d) result without enhancement; (e) result by Wang’s 
method; (f) result by our method. 


(a) (b) 



(e) (f) 


Fig. 4. IR and visible image fusion of image pair NV7404: (a) visible image; (b) IR 
image; (c) reference image; (d) result without enhancement; (e) result by Wang’s 
method; (f) result by our method. 

fused image by our method is intense red in color. Further more, 
places where the intensities of the IR image are much lower than 
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the background take on cyan, which will help judge the tempera¬ 
ture distribution of the scene, though there is no cold target. 


5. Discussion 

Human visual inspection will be a good choice to evaluate fused 
images as it is in accordance with the application of image fusion. 
However, the reproduction of subjective tests is often time-con¬ 
suming and difficult, while the exact same conditions for the test 
can not be guaranteed. Researchers tend to evaluate fused images 
with metrics and several well-accepted evaluation metrics for gray 
image fusion have been proposed [18-20]. Unfortunately, to the 
best of our knowledge, there is no effective and well-accepted 
objective metric for color fusion schemes. However, as we aim at 
improving the detectability of the target by enhancing color con¬ 
trast between the target and the background, we can roughly eval¬ 
uate different methods through their abilities of target-background 
separation [21]. As a result, we first measure the improvement in 
target detection by thresholding the color fused image through 
hue and saturation components [6,21] and obtaining a receiver- 
operator-characteristic (ROC) curve, which is a plot of false alarm 
rate versus detection rate. Then we measure the area-to-area 
[22] color distances between the target and the background for dif¬ 
ferent fusion methods. 

To obtain ROC curves, one precondition is to know the ideal tar¬ 
get detection result, which is obtained manually in this paper. Take 
the NV1816 pair for example, we firstly segment the image into 
three regions as shown in Fig. 5, where the indexes H, B and C stand 
for the hot target, the background and the cold target, respectively. 

According to the fact that color is perceived as intensity, hue 
and saturation by the human visual system, we perform target 



detection through the hue and saturation components in the chro¬ 
matic plane of the fused image: 


f 1, IH(x,y) - Ho| < H Th and |S(x,y) - 5 0 | < S Th , 

10, otherwise, 


(5) 


where H(x, y) and S(x, y) are the hue and saturation components of 
the fused image at the coordinate (x,y), respectively; H 0 and S 0 are 
the mean of the hue and saturation of the target, respectively; H Th 
and S Th are some given hue and saturation thresholds; and g(x,y) 
is the target detection result. The ROC curves are obtained with 
varying hue and saturation thresholds for the target detection, as 
shown in Fig. 6. 

From Fig. 6, it can be easily seen that the proposed method pro¬ 
vides an improvement for both the hot and the cold target detec¬ 
tion, as for the same false alarm rate, it obtains a higher 
detection rate. Wang’s method provides a modest performance 
improvement for the hot target detection. However, for the cold 
target detection, the performance by Wang’s method is not as good 
as that of the general fusion scheme without enhancement. 

The hit rate of color vision in detecting a target relies mainly on 
its ability to characterize color [23]. When the target is very close 
to the background in color, it results in a growth of false alarm rate. 
The effect of color contrast enhancement can be illustrated by 
comparing area-to-area color distances between the target region 
and the background for different fusion methods. As HSV color 
space is nonuniform, we still measure the color distance in YUV 
space: 

Dt-b = \J (Y T - Yb) 2 + (Ut - Ub ) 2 + {V T - V B ) 2 , ( 6 ) 

where the indexes T and B stand for the target and the background 
region, respectively. 

Table 1 shows the comparison of color distances in YUV space 
for different methods. Wang’s method enlarges the distance be¬ 
tween the hot target and the background moderately, while the 
distance provided by our method is more than twice that of the 
general fusion scheme without enhancement. Besides, our method 
enlarges the color distance between the cold target and the back¬ 
ground, resulting in a performance improvement demonstrated 
by the ROC curves for the cold target detection. 

The effectiveness of the proposed method can be further ex¬ 
plained through the enhancement ratios for each region. The mean 
enhancement ratios for all three regions by Wang’s and our meth¬ 
od are listed in Table 2. Both methods have the good property that 
they keep the background well as the mean enhancement ratios for 
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Fig. 6. ROC curves for the target detection with different image fusion methods: (a) ROC curves for the hot target detection; (b) ROC curves for the cold target detection. 
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Table 1 

Comparison of color distances in YUV space for different fusion methods. 


Color distance 

Without enhancement 

Wang’s method 

Our method 

Dh-b 

49.5 

60.3 

113.7 

Dc-b 

46.5 

45.6 

54.8 


Table 2 

The enhancement ratios for all three regions. 

Region 

Wang’s method 

Our method 

H 

2.34 

5.87 

C 

0.33 

2.95 

B 

1.01 

0.96 


the background tend to one. And for both methods, the mean 
enhancement ratios for the hot target region are larger than one, 
indicating that the hot target can be enhanced. However, as the 
enhancement ratio by our method is larger than that of Wang’s 
method, the proposed method can provide a better enhancement 
result. The mean enhancement ratio for the cold target by Wang’s 
method is much smaller than one, indicating that the cold target 
will be weakened in the fused image. In contrary, our method en¬ 
hances the cold target effectively as the mean enhancement ratio 
for the cold target region is much larger than one. 

6. Conclusion 

An improved color image-fusion scheme is presented by intro¬ 
ducing a color contrast enhancement approach, which gives boost 
to color of both hot and cold targets in the final fused image. Color 
contrast is effectively enhanced in our IR and visible image-fusion 
scheme, where hot targets will appear intense red and cold targets 
will appear cyan. The special relationship between targets and col¬ 
ors in the fused image offers more temperature information to the 
observer, which can aid target detection and improve observer per¬ 
formance. The proposed color contrast enhancement ratio is in 
accordance with the characteristics of the target and the back¬ 
ground in IR images. The enhancement ratios for both hot and cold 
targets are larger than one, while it tends to one for the back¬ 
ground. Thus both hot and cold targets can be enhanced, while de¬ 
tails of the background are well kept by the proposed method, 
indicating the method is suitable for both target detection and sit¬ 
uation awareness tasks. 

The method is efficient in implementation. With a Pentium(R) 4 
CPU of 2.8G Hz and 512 M RAM, experiments are carried out with 
Matlab 7.0. For the uncamp set tested, 3.1 s is spent to complete the 
fusion of all 32 pair of images with the resolution 360 x 270. That 
is to say, 10 frames per second can be achieved with modest 
requirement, indicating that real time implementation can be ex¬ 
pected with appropriate hardware design. 

The proposed color contrast enhancement approach can be fur¬ 
ther generalized in image-fusion schemes that utilize other color 
space. For example, Toet’s fusion algorithm in loc/3 color space [7] 
can be improved by enhancing a channel using the enhancement 
ratio proposed in this paper. Further more, better visual results 


may be obtained when combined with local algorithms such as 
in [9], 
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